Use of Static Surrogates in Hyperparameter Optimization

Optimizing the hyperparameters and architecture of a neural network is long yet necessary phase in most applications. This consuming process can benefit from strategies designed to discard low-quality configurations quickly focus on more promising candidates. work aims at enhancing HyperNOMAD, library that adapts direct search derivative-free optimization algorithm tune both training simultaneously. Two static surrogates are developed trigger an early stopping during configuration evaluation strategically rank pool These additions HyperNOMAD shown reduce its resource consumption by orders magnitude without harming quality proposed solutions.